Using Large Language Models to Support Content Analysis: A Case Study of ChatGPT for Adverse Event Detection.

This study explores the potential of using large language models to assist content analysis by conducting a case study to identify adverse events (AEs) in social media posts. The case study compares ChatGPT's performance with human annotators' in detecting AEs associated with delta-8-tetrahydrocannabinol, a cannabis-derived product. Using the identical instructions given to human annotators, ChatGPT closely approximated human results, with a high degree of agreement noted: 94.4% (9436/10,000) for any AE detection (Fleiss κ=0.95) and 99.3% (9931/10,000) for serious AEs (κ=0.96). These findings suggest that ChatGPT has the potential to replicate human annotation accurately and efficiently. The study recognizes possible limitations, including concerns about the generalizability due to ChatGPT's training data, and prompts further research with different models, data sources, and content analysis tasks. The study highlights the promise of large language models for enhancing the efficiency of biomedical research.

Characteristics of X (Formerly Twitter) Community Notes Addressing COVID-19 Vaccine Misinformation.

This study evaluated the topics, accuracy, and credibility of X (formerly Twitter) Community Notes addressing COVID-19 vaccination.

Correction: Twitter and Facebook posts about COVID-19 are less likely to spread misinformation compared to other health topics.

[This corrects the article DOI: 10.1371/journal.pone.0261768.].

Twitter social mobility data reveal demographic variations in social distancing practices during the COVID-19 pandemic.

The COVID-19 pandemic demonstrated the importance of social distancing practices to stem the spread of the virus. However, compliance with public health guidelines was mixed. Understanding what factors are associated with differences in compliance can improve public health messaging since messages could be targeted and tailored to different population segments. We utilize Twitter data on social mobility during COVID-19 to reveal which populations practiced social distancing and what factors correlated with this practice. We analyze correlations between demographic and political affiliation with reductions in physical mobility measured by public geolocation tweets. We find significant differences in mobility reduction between these groups in the United States. We observe that males, Asian and Latinx individuals, older individuals, Democrats, and people from higher population density states exhibited larger reductions in movement. Furthermore, our study also unveils meaningful insights into the interactions between different groups. We hope these findings will provide evidence to support public health policy-making.

Managing HIV During the COVID-19 Pandemic: A Study of Help-Seeking Behaviors on a Social Media Forum.

Although numerous editorials claim the COVID-19 pandemic has disproportionately impacted vulnerable populations, particularly those affected by HIV, these claims have received limited empirical evaluation. We analyzed posts to Reddit's r/HIVAIDS from January 3, 2012 through April 30, 2022 to (a) assess changes in the volume of posts during the pandemic and (b) determine the needs of HIV affected communities. There were cumulatively 100% (95%CI: 75-126) more posts than expected since the US declared a pandemic emergency. The most prevalent themes in these posts were for obtaining an HIV + diagnosis (representing 34% (95%CI:29-40) of all posts), seeking HIV treatment (20%; 95%CI:16-25), finding psychosocial support (16%; 95%CI:12-20), and tracking disease progression (8%; 95%CI:5-11). Discussions about PrEP and PEP were the least common, representing less than 6% of all posts each. Social media has increasingly become an important health resource for vulnerable populations seeking information, advice, and support. Public health organizations should recognize how the lay public uses social media and collaborate with social media companies to ensure that the needs of help-seekers on these platforms are met.

Evaluating Artificial Intelligence Responses to Public Health Questions.

This cross-sectional study analyzes the quality of ChatGPT responses to public health questions.

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.

Importance: The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians. Objective: To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions. Design, Setting, and Participants: In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians. Results: Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P < .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot. Conclusions: In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

Twitter and Facebook posts about COVID-19 are less likely to spread misinformation compared to other health topics.

The COVID-19 pandemic brought widespread attention to an "infodemic" of potential health misinformation. This claim has not been assessed based on evidence. We evaluated if health misinformation became more common during the pandemic. We gathered about 325 million posts sharing URLs from Twitter and Facebook during the beginning of the pandemic (March 8-May 1, 2020) compared to the same period in 2019. We relied on source credibility as an accepted proxy for misinformation across this database. Human annotators also coded a subsample of 3000 posts with URLs for misinformation. Posts about COVID-19 were 0.37 times as likely to link to "not credible" sources and 1.13 times more likely to link to "more credible" sources than prior to the pandemic. Posts linking to "not credible" sources were 3.67 times more likely to include misinformation compared to posts from "more credible" sources. Thus, during the earliest stages of the pandemic, when claims of an infodemic emerged, social media contained proportionally less misinformation than expected based on the prior year. Our results suggest that widespread health misinformation is not unique to COVID-19. Rather, it is a systemic feature of online health communication that can adversely impact public health behaviors and must therefore be addressed.

Demographic Representation and Collective Storytelling in the Me Too Twitter Hashtag Activism Movement.

The #MeToo movement on Twitter has drawn attention to the pervasive nature of sexual harassment and violence. While #MeToo has been praised for providing support for self-disclosures of harassment or violence and shifting societal response, it has also been criticized for exemplifying how women of color have been discounted for their historical contributions to and excluded from feminist movements. Through an analysis of over 600,000 tweets from over 256,000 unique users, we examine online #MeToo conversations across gender and racial/ethnic identities and the topics that each demographic emphasized. We found that tweets authored by white women were overrepresented in the movement compared to other demographics, aligning with criticism of unequal representation. We found that intersected identities contributed differing narratives to frame the movement, co-opted the movement to raise visibility in parallel ongoing movements, employed the same hashtags both critically and supportively, and revived and created new hashtags in response to pivotal moments. Notably, tweets authored by black women often expressed emotional support and were critical about differential treatment in the justice system and by police. In comparison, tweets authored by white women and men often highlighted sexual harassment and violence by public figures and weaved in more general political discussions. We discuss the implications of this work for digital activism research and design, including suggestions to raise visibility by those who were under-represented in this hashtag activism movement. Content warning: this article discusses issues of sexual harassment and violence.

#MeToo and Google Inquiries Into Sexual Violence: A Hashtag Campaign Can Sustain Information Seeking.

The #MeToo Movement has brought new attention to sexual harassment and assault. While the movement originates with activist Tarana Burke, actor Alyssa Milano used the phrase on Twitter in October 2017 in response to multiple sexual harassment allegations against Hollywood producer Harvey Weinstein. Within 24 hours, 53,000 people tweeted comments and/or shared personal experiences of sexual violence. The study objective was to measure how information seeking via Google searches for sexual harassment and assault changed following Milano's tweet and whether this change was sustained in spite of celebrity scandals. Weekly Google search inquiries in the United States were downloaded for the terms metoo, sexual assault, sexual harassment, sexual abuse, and rape for January 1, 2017 to July 15, 2018. Seven related news events about perpetrator accusations were considered. Results showed that searches for metoo increased dramatically after the Weinstein accusation and stayed high during subsequent accusations. A small decrease in searches followed, but the number remained very high relative to baseline (the period before the Weinstein accusation). Searches for sexual assault and sexual harassment increased substantially immediately following the Weinstein accusation, stayed high during subsequent accusations, and saw a decline after the accusation of Matt Lauer (talk show host; last event considered). We estimated a 40% to 70% reduction in searches 6 months after the Lauer accusation, though the increase in searches relative to baseline remained statistically significant. For sexual abuse and rape, the number of searches returned close to baseline by 6 months. It appears that the #MeToo movement sparked greater information seeking that was sustained beyond the associated events. Given its recent ubiquitous use in the media and public life, hashtag activism such as #MeToo can be used to draw further attention to the next steps in addressing sexual assault and harassment, moving public web inquiries from information seeking to action.

News coverage of the E-cigarette, or Vaping, product use Associated Lung Injury (EVALI) outbreak and internet searches for vaping cessation.

BACKGROUND: In the latter half of 2019, an outbreak of pulmonary disease in the USA resulted in 2807 hospitalisations and 68 deaths, as of 18 February 2020. Given the severity of the outbreak, we assessed whether articles during the outbreak era more frequently warned about the dangers of vaping and whether internet searches for vaping cessation increased. METHODS: Using Tobacco Watcher, a media monitoring platform that automatically identifies and categorises news articles from sources across the globe, we obtained all articles that (a) discussed the outbreak and (b) primarily warned about the dangers of vaping. We obtained internet search trends originating from the USA that mentioned 'quit' or 'stop' and 'e cig(s),' 'ecig(s),' 'e-cig(s),' 'e cigarette(s),' 'e-cigarette(s),' 'electronic cigarette(s),' 'vape(s),' 'vaping' or 'vaper(s)' from Google Trends (eg, 'how do I quit vaping?'). All data were obtained from 1 January 2014 to 18 February 2020 and ARIMA models were used with historical trends to forecast the ratio of observed to expected search volumes during the outbreak era. RESULTS: News of the vaping-induced pulmonary disease outbreak was first reported on 25 July 2019 with 195 articles, culminating in 44 512 articles by 18 February 2020. On average, news articles warning about the dangers of vaping were 130% (95% prediction interval (PI): -15 to 417) and searches for vaping cessation were 76% (95% PI: 28 to 182) higher than expected levels for the days during the period when the sources of the outbreak were unknown (25 July to 27 September 2019). News and searches stabilised just after the US Centers for Disease Control and Prevention reported that a primary source of the outbreak was an additive used in marijuana vapes on 27 September 2019. In sum, there were 12 286 articles archived in Tobacco Watcher primarily warning about the dangers of vaping and 1 025 000 cessation searches following the outbreak. CONCLUSION: The vaping-induced pulmonary disease outbreak spawned increased coverage about the dangers of vaping and internet searches for vaping cessation. Resources and strategies that respond to this elevated interest should become a priority among public health leaders.

How Internet Contracts Impact Research: Content Analysis of Terms of Service on Consumer Product Websites.

BACKGROUND: Companies use brand websites as a promotional tool to engage consumers on the web, which can increase product use. Given that some products are harmful to the health of consumers, it is important for marketing associated with these products to be subject to public health surveillance. However, terms of service (TOS) governing the use of brand website content may impede such important research. OBJECTIVE: The aim of this study is to explore the TOS for brand websites with public health significance to assess possible legal and ethical challenges for conducting research on consumer product websites. METHODS: Using Statista, we purposefully constructed a sample of 15 leading American tobacco, alcohol, psychiatric pharmaceutical, fast-food, and gun brands that have associated websites. We developed and implemented a structured coding system for the TOS on these websites and coded for the presence versus absence of different types of restriction that might impact the ability to conduct research. RESULTS: All TOS stated that by accessing the website, users agreed to abide by the TOS (15/15, 100%). A total of 11 out of 15 (73%) websites had age restrictions in their TOS. All alcohol brand websites (5/15, 33%) required users to enter their age or date of birth before viewing website content. Both websites for tobacco brands (2/15, 13%) further required that users register and verify their age and identity to access any website content and agree that they use tobacco products. Only one website (1/15, 7%) allowed users to display, download, copy, distribute, and translate the website content as long as it was for personal and not commercial use. A total of 33% (5/15) of TOS unconditionally prohibited or put substantial restrictions on all of these activities and/or failed to specify if they were allowed or prohibited. Moreover, 87% (13/15) of TOS indicated that website access could be restricted at any time. A total of 73% (11/15) of websites specified that violating TOS could result in deleting user content from the website, revoking access by having the user's Internet Protocol address blocked, terminating log-in credentials, or enforcing legal action resulting in civil or criminal penalties. CONCLUSIONS: TOS create complications for public health surveillance related to e-marketing on brand websites. Recent court opinions have reduced the risk of federal criminal charges for violating TOS on public websites, but this risk remains unclear for private websites. The public health community needs to establish standards to guide and protect researchers from the possibility of legal repercussions related to such efforts.

Facebook Pages, the "Disneyland" Measles Outbreak, and Promotion of Vaccine Refusal as a Civil Right, 2009-2019.

Objectives. To understand changes in how Facebook pages frame vaccine opposition.Methods. We categorized 204 Facebook pages expressing vaccine opposition, extracting public posts through November 20, 2019. We analyzed posts from October 2009 through October 2019 to examine if pages' content was coalescing.Results. Activity in pages promoting vaccine choice as a civil liberty increased in January 2015, April 2016, and January 2019 (t[76] = 11.33 [P < .001]; t[46] = 7.88 [P < .001]; and t[41] = 17.27 [P < .001], respectively). The 2019 increase was strongest in pages mentioning US states (t[41] = 19.06; P < .001). Discussion about vaccine safety decreased (rs[119] = -0.61; P < .001) while discussion about civil liberties increased (rs[119] = 0.33; Py < .001]). Page categories increasingly resembled one another (civil liberties: rs[119] = -0.50 [P < .001]; alternative medicine: rs[84] = -0.77 [P < .001]; conspiracy theories: rs[119] = -0.46 [P < .001]; morality: rs[106] = -0.65 [P < .001]; safety and efficacy: rs[119] = -0.46 [P < .001]).Conclusions. The "Disneyland" measles outbreak drew vaccine opposition into the political mainstream, followed by promotional campaigns conducted in pages framing vaccine refusal as a civil right. Political mobilization in state-focused pages followed in 2019.Public Health Implications. Policymakers should expect increasing attempts to alter state legislation associated with vaccine exemptions, potentially accompanied by fiercer lobbying from specific celebrities.

Adapting and Extending a Typology to Identify Vaccine Misinformation on Twitter.

Objectives. To adapt and extend an existing typology of vaccine misinformation to classify the major topics of discussion across the total vaccine discourse on Twitter.Methods. Using 1.8 million vaccine-relevant tweets compiled from 2014 to 2017, we adapted an existing typology to Twitter data, first in a manual content analysis and then using latent Dirichlet allocation (LDA) topic modeling to extract 100 topics from the data set.Results. Manual annotation identified 22% of the data set as antivaccine, of which safety concerns and conspiracies were the most common themes. Seventeen percent of content was identified as provaccine, with roughly equal proportions of vaccine promotion, criticizing antivaccine beliefs, and vaccine safety and effectiveness. Of the 100 LDA topics, 48 contained provaccine sentiment and 28 contained antivaccine sentiment, with 9 containing both.Conclusions. Our updated typology successfully combines manual annotation with machine-learning methods to estimate the distribution of vaccine arguments, with greater detail on the most distinctive topics of discussion. With this information, communication efforts can be developed to better promote vaccines and avoid amplifying antivaccine rhetoric on Twitter.